Skip to content

chore(metrics): freeze the proc-v1 process digests - #991

Open
modelmirror wants to merge 2 commits into
stagingfrom
chore/process-digest-freeze
Open

chore(metrics): freeze the proc-v1 process digests#991
modelmirror wants to merge 2 commits into
stagingfrom
chore/process-digest-freeze

Conversation

@modelmirror

Copy link
Copy Markdown
Collaborator

What

The #907 freeze cutover, as two commits so the record can name the paste:

  1. 8bf09ed2 pastes the six digests from fedcourts process-digest --all into FROZEN_PROCESS_DIGESTS (three predictors, three evaluators, all proc-v1), and converts the shakedown-emptiness test into the frozen-state test — the set must resolve exactly against the current registry, so any post-freeze prompt/config drift fails the suite instead of riding silently.
  2. Records that commit in docs/milestones.md as the cutover marker.

Merge with a merge commit, not squash — squashing rewrites the sha the record names.

Pre-freeze evidence

  • All-scenario integration suite green twice tonight at the promoted content (staging 4b93895e and 802ea80e), including engine-smoke × claude-code/codex/gemini — real-engine predict cells on the staging keys.
  • Outstanding from metrics: bless the settled process digest into the frozen set and record the freeze commit #907's checklist: real-engine evaluate cells for codex and gemini. The default smoke case has an open event (predict-only); an evaluate-exercising smoke needs a resolved case, e.g.:
    gh workflow run integration-test.yml --ref staging -f scenario=engine-smoke -f engine=codex -f docket=73129750 (and -f engine=gemini). Recommend running both before merging this.

Effect

None until promoted: is_frozen is read by leaderboard/ops-report from the default branch, so the freeze takes effect at the next (small) staging→main promotion — which closes #907 and the OT2026 first-conference milestone alongside #864.

Closes #907 (hand-close on the promotion that carries this, per the freeze-takes-effect-on-main rule).

Merge lane

Maintainer's call — mechanically src+tests+docs, but it IS the pre-registration cutover; leaving the merge to the maintainer deliberately.

modelmirror and others added 2 commits August 2, 2026 02:27
The freeze cutover: FROZEN_PROCESS_DIGESTS takes the digest of every enabled
predictor and evaluator as read off `fedcourts process-digest --all` at the
promoted state. The shakedown-emptiness test becomes the frozen-state test: the
set must resolve exactly against the current registry, so post-freeze drift in
a prompt or registry config surfaces as a mismatch here.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Copy link
Copy Markdown
Collaborator Author

Holding this merge deliberately — it is the freeze cutover, and the pre-freeze queue now assigned to the OT2026 milestone (#897, #903, #904, #963, #967, plus #864's remaining verification) lands first. Two notes for whoever merges it:

  1. Re-paste the digests before merging if any of the queued work moves a prompt or config that feeds compute_process_digestmetrics: decide how segment_base_rate reconciles a sal-v2 band against a lagging statpack #897 (evaluate.md's segment_base_rate text), evaluate: score predicted judicial reasoning (two-tier) #903/analytics: surface predicted-reasoning scores, and validate the judge against Tier 1 #904 (a claim set touches both prompts), and salience: sal-v1 conditions on signals a petition does not have on arrival — rework for prospective use #963 (predict.md names sal-v1 verbatim) all can. Re-run fedcourts process-digest --all at the final content and update the FROZEN_PROCESS_DIGESTS paste; the second commit's milestones.md sha reference must then also be redone (which is fine — merge commit, not squash, per the description).
  2. The "closes … alongside metrics: prove the leaderboard consume path on production evaluations #864" line is premature. The frozen-scope leaderboard is currently empty (entries: [], evaluations_total: 0), the evaluate channel is paused with 3 queued events held by the handoff variable, and mean_brier_skill_score has never produced a number. metrics: prove the leaderboard consume path on production evaluations #864's own rescope comment names what remains: resume the handoff, let the three held cells grade, hand-check each segment_base_rate against the pre-computed sheet. metrics: prove the leaderboard consume path on production evaluations #864 stays open past this promotion until that check is done — it is itself a pre-freeze validation of the metric the freeze makes claimable.

Copy link
Copy Markdown
Collaborator Author

The pasted digests are stale — do not merge this as it stands

All six values in FROZEN_PROCESS_DIGESTS predate the pre-freeze batch. Two merged changes moved every actor's digest: #1013 (the cert-v1 claim contract in predict.md / evaluate.md) and #1024 (the redaction change), plus #1022 and #1027 (the stage-conditional prompt sections). The digest is sha256(prompt_bytes ␀ canonical{engine, resolved model, prompt_path, retrieval, mcp_servers}), so a prompt edit moves it by construction.

The PR's own frozen-state test is the backstop here — it requires the set to resolve exactly against the current registry, so merging the stale paste fails the suite rather than silently freezing the wrong process. Worth stating anyway, because the failure would read as a broken test rather than a stale paste.

fedcourts process-digest --all on the current staging tip (63bef3b3) produces:

proc-v1  predictor  claude-baseline  sha256:30c4b708a2a28e608aa906858758aa2cfe972c3b36a0dd15952f6c1db4f9775d
proc-v1  predictor  codex-baseline   sha256:b80f2290bfb29c12801e84d9b4a60dc992426a7a9eb91b85dd0a7c1835968ed8
proc-v1  predictor  gemini-baseline  sha256:928552054ef6deb61dbf1f7f5648a7eb66f0bfd4b52d646aa9746f69a29fecf2
proc-v1  evaluator  claude-judge     sha256:7ed1a24d1b4cc7fea57f229a29ecc700e980eccd7e2d2b7b071412a343581c4a
proc-v1  evaluator  codex-judge      sha256:2fe642cf0a84065c129241bdd053efdb4e243b524b78b8cfe747e3953ef15047
proc-v1  evaluator  gemini-judge     sha256:56c9cfd91024512ee24af3fe34f312dc0e279b602e28b9e0a0e3b8f443512b77

These are a preview, not the paste. sal-v2 (#963) is still to land and the promotion has not happened, so the sequence stays: land sal-v2 → promote (#1019, merge commit) → run fedcourts process-digest --all on the promoted tree → paste that output here → merge-commit → bless.

The "Pre-freeze evidence" section above is also stale: the all-scenario green it cites was at staging 4b93895e / 802ea80e, many merges back. The evaluate-side engine smoke it lists as outstanding is still outstanding.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant